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ABSTRACT 

This study reports on an effort to illustrate the coupling of educational research with ongoing curriculum development to 
promote effective and evidence-based online learning. The research findings have been used to inform the EarthLabs 
curriculum development team as they revise existing modules and create new modules, in order to represent the ways in 
which such research findings can be used to improve similar online curriculum materials and enhance student learning 
outcomes. EarthLabs curriculum is a suite of online inquiry-based activities that promote understanding of Earth system 
science. Assessments were employed to understand student learning about complex climate systems as a result of their 
engagement with the online EarthLabs curriculum. Collection of pre- and postcourse student assessment data ( n = 205), 
classroom observations during implementation ( n = 6), teacher interviews ( n = 7), and eye-tracking data (n = 49) were 
included in the study. Qualitative and quantitative findings show that EarthLabs classroom implementation significantly 
improves students' conceptual and systems understanding and that students and external users are appropriately engaged 
with the online materials. These findings have been applied to evaluating the efficacy of the EarthLabs program in reaching 
target programmatic and learning goals, as well as to developing a broader understanding of the cognitive challenges students 
have in navigating complex Earth systems phenomena, where continued EarthLabs program revision has occurred through 
design-based research. © 2014 National Association of Geoscience Teachers. [DOI: 10.5408/13-060.1] 
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INTRODUCTION 

Humans are modifying Earth without fully understand¬ 
ing how our actions affect the planet's major systems: 
atmosphere (air), hydrosphere (water), biosphere (life), and 
geosphere (land). Research indicates that increases in 
globally averaged temperatures of just a few degrees in this 
century will likely cause an increase in the occurrence of 
drought, floods, and extreme weather, and accelerate sea- 
level rise into the future (IPCC, 2007a, 2007b, 2007c, 2013). 
The inclusion of climate literacy in public education is 
necessary for society to develop strategies to address climate 
change, yet climate literacy demands cognitive and percep¬ 
tual leaps for students and teachers (Grotzer and Lincoln, 
2007) with respect to understanding complex interactions 
among the components of the Earth system. Understanding 
climate change requires grasping complex interactions 
among the atmosphere, hydrosphere (including the cryo- 
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sphere), biosphere, and geosphere on multiple spatial and 
temporal scales. In a sense, climate change offers our most 
compelling context for helping students to learn Earth 
systems science and develop essential scientific thinking 
skills. Further, scientific practices and the essential thinking 
skills, such as modeling, have now been incorporated into 
the Next Generation Science Standards (NGSS), and thus 
their importance has been recognized in the larger context of 
science education (National Research Council, 2012; 
Achieve, Inc., 2013). 

In parallel with the learning challenges that accompany 
the complexities of climate and Earth systems, we must keep 
in mind that remotely sensed data, model predictions, and 
ongoing direct observations continually change our under¬ 
standing of climate. In a field where the most current and 
up-to-date science is critical for understanding the state of 
the problem, the flexibility of an online curriculum has 
benefits over static textbooks. Online materials allow 
students to access current science through near real-time 
data and use of authentic technological tools that support 
the perceptual and conceptual challenges learners may have 
with the unique features of the climate system (e.g., 
visualizing the temporal and spatial dynamics). Many online 
lessons and curricula are available to educators, including 
some that address climate phenomena. This wealth of 
resources makes it difficult for educators to identify 
empirically tested materials, understand them well enough 
to utilize them with students, and know which ones will be 
most effective in supporting their classroom learning 
objectives. As such, recent efforts to assist educators in 
easily accessing peer-reviewed materials have been made, 
including the Climate and Energy Awareness Network 
(CLEAN) (Gold et al., 2012) and the Diversity and 
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Innovation in the Geosciences (DIG) Texas Project (Ellins et 
al., 2014). The EarthLabs modules have served as a pilot case 
in best practice for reviewing units and modules for the 
CLEAN Collection as an evidence-based curriculum that can 
be implemented "as is" in the classroom. 

In order to meet student learning needs through 
effective curriculum, the coupling of education research with 
curriculum design is needed (Design-Based Research Col¬ 
lective, 2003). Design-based research (or DBR), originally 
developed by Brown (1992), has been gaining traction in the 
literature and is based on the following characteristics (Wang 
and Hannafin, 2005; Anderson and Shattuck, 2012): (1) It 
should design and test a significant intervention. (2) It 
should be practical, where the research refines both theory 
and practice. (3) It should be grounded in theory to inform 
the research design. (4) It should be iterative, interactive, and 
flexible, where designers are involved in the process and 
work together with participants, and iterative cycles of 
analysis, design, implementation, and redesign are used. (5) 
It should be integrative, where mixed method approaches 
are used in the research. (6) Finally, it should be contextual, 
where the research results are connected with the design 
process and have a practical impact on instructional practice 
within specific settings. The advantages of DBR are that it is 
tied to practice, creating opportunities for novel learning and 
teaching environments, raises important questions for 
continued research, contributes to theories about learning 
and teaching, advances and consolidates design knowledge, 
and increases capacity for educational innovation (Edelson, 
2002; Design-Based Research Collective, 2003). DBR has also 
been described as especially applicable to technology- 
enhanced learning environments (Edelson et al., 1999; Linn 
et al., 2004; Sandoval and Reiser, 2004; Wang and Reeves, 
2006), such as the EarthLabs project described herein. The 
project described in this study is part of a larger project 
(Ledley et al., 2012) that includes curriculum development, 
teacher training, and a research program with the program¬ 
matic goals of: (1) developing informed teachers confident in 
using the EarthLabs online curriculum; (2) creating curricu¬ 
lum that has been informed by master teachers and tested 
with students; (3) investigating how the curriculum supports 
student conceptual understanding of the climate system; (4) 
understanding how students engage with the EarthLabs 
curriculum so that it can effectively meet their learning 
needs; and (5) considering the ways in which the EarthLabs 
curriculum should be modified to address research findings. 
This study will focus on items 3-5. Item 2 will be addressed 
in further detail in a companion paper in this special issue 
(Ellins et al., 2014). 


COGNITIVE CHALLENGES OF SYSTEMS 
THINKING 

Climate change has been shown to be a difficult concept 
for learners at all levels to understand fully, from decision¬ 
making adults (Leiserowitz, 2008) to young children (Francis 
et al., 1993). Although noncognitive variables, such as 
political and religious orientation (McCright and Dunlap, 
2011; Leiserowitz, 2008), can influence perceptions of 
climate change, basic conceptual understanding is likely 
significantly influenced by an individual's ability to engage in 
"systems thinking" (Assaraf and Orion, 2004; Gautier and 
Rebich, 2005; Rebich and Gautier, 2005; Hmelo-Silver et al.. 


2007). Grasping the nature of systems is vital to developing 
coherent mental models of climate. 

A student's ability to reason about complex Earth 
phenomena depends upon how well new ideas are 
integrated with preexisting mental models (Vosniadou and 
Brewer, 1992; Chi, 2005). In the study by Sell et al. (2006), 
most college students were unable to articulate processes 
that are key to how Earth changes over time and space, 
suggesting a limited ability to understand Earth system 
science and to think across spatial and temporal dimensions. 
Research into student conceptions about Earth (Vosniadou 
and Brewer, 1992; Dove, 1998; Blake, 2005; Libarkin, 2006; 
Libarkin and Kurdzeil, 2006) shows that students have a 
range of nonscientific ideas about how Earth changes over 
time and space. Studies of student understanding of geologic 
time (Ault, 1982; Schoon, 1992; Oversby, 1996; Marques and 
Thompson, 1997; DeLaughter et al., 1998; Trend, 2001; 
Dodick and Orion, 2003; Dahl et al., 2005; Libarkin and 
Anderson, 2005) suggest that students are more comfortable 
with relative time than absolute, perhaps because people of 
all ages have difficulty comprehending differences between 
large numbers and larger numbers (e.g., thousands vs. 
millions; Greeno, 1991; Libarkin and Anderson, 2005). 

Systems thinking requires recognition that observed 
phenomena result from underlying processes, and that these 
processes can interact to produce complex phenomena. 
Systems thinking also requires one to understand that not all 
interactions are purely linear (Herbert, 2005; Raia, 2005). In 
the climate system, for example, positive and negative 
feedback loops generate fluctuations in temperature that 
may not be obvious from initial inspection of the system. 
Another possible explanation for conceptual difficulties with 
climate change results from its multidisciplinary nature (Sell 
at al., 2006), such that understanding climate change 
requires crossing boundaries between geology and geogra¬ 
phy, physics and chemistry, and atmospheric and ocean 
sciences. 

Research evidence suggests that the use of multiple 
representations that include technology and hands-on 
activities (McNeal et al., 2008), explicit identification of 
spatial characteristics of phenomena (Black, 2005), and 
discussion of the impact of climate change on society 
(Gautier and Rebich, 2005) may effectively assist students' 
conceptual understanding of complex Earth processes. 
Edelson's (2001) work with the online software platform 
"My World GIS" demonstrated how authentic scientific 
tools and technology can be used to support students' spatial 
understanding. Furthermore, research applying the use of 
hypermedia to complex systems has been shown to be 
effective 0acobson, 2008; Liu and Hmelo-Silver, 2009) while 
also increasing student motivations towards science (Wang 
and Reeves, 2006). 

With the increase in affordability and availability of 
technology tools, Web-based learning has been amplified in 
the K-12 classroom (Picciano and Seaman, 2009), while 
climate change and Earth system concepts have also 
increased in their prevalence in national science standards 
(National Research Council, 2012; Achieve, Inc., 2013). 
However, evidence-based online curriculum that employs 
educational research to specifically address the K-12 
classroom (Means et al., 2010), teacher training in technol¬ 
ogy use in the classroom (Kleiman, 2000) and climate 
change content (Sweeney and Sterman, 2007), and curric- 
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ulum units that utilize current pedagogical approaches such 
as inquiry-based activities, constructivist learning principles, 
and proper evaluation methods are needed (Mioduser et al., 
2000). These needs are especially important given the large 
amount of and vastly growing collection of online climate 
change materials available to educators (for examples, see 
http://cleanet.org and http://www.camelclimatechange.org). 

RESEARCH QUESTIONS: STUDENT 
UNDERSTANDING AND ENGAGEMENT WITH 
COMPLEX SYSTEMS 

This work evaluates the efficacy of online curriculum 
materials through a research design that embeds pre- and 
postcourse student testing, teacher interviews, classroom 
observations, and eye-tracking studies applied to the case of 
the online EarthLabs project. The EarthLabs program 
includes the development of Earth Science and environ¬ 
mental science curriculum materials that engage high school 
students in a combination of Web-based activities, hands-on 
experiments, and scientific data analysis, with the ultimate 
goal of providing easily accessible, inexpensive, and effective 
inquiry-based experiences (Ledley et al., 2012). In this paper, 
we report on student understanding of and engagement 
with complex Earth systems. Specifically, we sought to 
answer the questions: (1) How does the online EarthLabs 
curriculum assist students in developing understanding of 
temporal and spatial dynamics and system interactions of 
climate? (2) How do users engage with and navigate the 
online EarthLabs curriculum? 


METHODS 

Curriculum Development 

The EarthLabs project (Ledley et al., 2012) addresses 
nine subjects using the following live modules: Hurricanes, 
Corals, Fisheries, Drought, Climate and the Cryosphere, Earth 
System Science, Climate and the Biosphere, Climate and the 
Carbon Cycle, and the Climate Detectives (teacher guide: 
http://serc.carleton.edu/EarthLabs; student portal: http:// 
serc.carleton.edu/eslabs). An EarthLabs module consists of 
five to nine sequenced labs intended to build on the 
knowledge and skills learned in the previous labs. Activities 
consist of a combination of online reading, data manipula¬ 
tion and visualization using software applications such as 
Google Earth and ImageJ, hands-on activities, and outdoor 
explorations. Each lab within an EarthLabs module contains 
"Checking In" and "Stop and Think" questions that assist 
students and teachers in gauging learning progress. The 
EarthLabs teacher portal provides teachers with relevant 
background information; lab overviews; lists of required 
materials, technical resources, and online tools; additional 
science content support; and suggestions for assessment and 
extension activities. 

In this paper, we focus on two of four modules that 
address climate and Earth systems, with emphasis on the 
classroom implementation and research surrounding the 
Earth System Science (ESS) and Climate and the Cryosphere 
(Cryosphere) modules. Feedback from the research results, 
including teacher professional development (Ellins et al., 
2014), of these two modules is currently being incorporated 
into the revision of all modules. 


The local to global scale approach to teaching and 
learning has been used effectively in geography education 
(Geography Education Standards Project, 1994). As such, the 
ESS module begins by having students examine Earth 
system processes on the temporal and spatial scales that 
are most familiar to them. The first few labs focus on the 
local scale—the scale that students experience every day, 
such as their schoolyard, a neighborhood park, or other 
common area within the local community. The next labs 
focus on the regional scale and emphasize the boundaries of 
a region and the interactions among components. The last 
few labs focus on the global scale and highlight circulation 
patterns, the water cycle, and change over time. The 
Cryosphere module similarly begins with the local and 
evolves into the global scale, and it focuses on snow and 
ice, melting and freezing processes, observations of how 
land and sea ice change over time, interactions and 
feedbacks that contribute to increased melting within the 
cryosphere, and how these processes relate to global climate. 
Each module contains seven labs and specific learning goals 
(Table I). The third and fourth modules. Climate and the 
Biosphere and Climate and the Carbon Cycle, have benefited 
from this effort by incorporating feedback from emerging 
research data. Climate and the Biosphere focuses on climate 
processes, the relationship and differences between weather 
and climate, and the impacts of weather and climate on 
Earth's biomes. Climate and the Carbon Cycle focuses on the 
role carbon plays in influencing climate. 

The EarthLabs Design-Based Research Approach 

We implemented an online curriculum with embedded 
student research and followed a create-test-revise-imple- 
ment-revise iterative process. This process incorporated the 
following elements: 

(1) Create. Curriculum developers designed explicit 
student learning goals and developed curriculum 
level assessments while online materials were 
generated, following modified backwards design 
(Wiggens and McTighe, 1998, 2005) procedures, and 
generating a framework for project researchers to 
align the evaluation and research components with 
the EarthLabs curriculum. 

(2) Test. Researchers developed appropriate research 
questions and coupled research quality assessments 
before and during the curriculum development 
process and worked with developers to ensure that 
assessment content was aligned with EarthLabs 
curriculum goals and materials. 

(3) Revise. Discussion between researchers and curric¬ 
ulum developers occurred continuously to generate 
feedback and continual revision of student learning 
goals, research questions, and developed assess¬ 
ments before curriculum implementation. 

(4) Implement. Appropriate qualitative and quantitative 
approaches were employed to address the research 
questions generated in order to triangulate data, 
enhance robustness of interpretations, and maxi¬ 
mize data collection opportunities. Validity and 
reliability measures were implemented. 

(5) Revise. Collected data and results were shared with 
curriculum developers well before upcoming imple¬ 
mentation stages so that adequate modifications 
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TABLE I: Student learning goals and labs for the Cryosphere and Earth Systems Science EarthLabs modules. 



Cryosphere 

Earth System Science 

Learning Goal 

Students will learn about the 
thermodynamic, dynamic, and feedback 
processes in the cryosphere 

Students will learn to identify the parts of the Earth system 
and the processes that connect them 

Students will Address 

What is the cryosphere? How and why 
does the cryosphere change over time and 
space? What are the timescales associated 
with changes in the cryosphere? How do 
climate and the cryosphere influence each 
other? 

What is Earth system science? How can we describe Earth 
as a system? How are energy and matter exchanged 
among the four main components of the Earth system 
(atmosphere, biosphere, hydrosphere, pedosphere)? How 
does the Earth system change over time? How is life 
affected by changes in the Earth system? 

Lab 1 

Frozen in Time 

Think Globally, Act Locally 

Lab 2 

Sea Ice Thermodynamics 

Drawing Local Connections 

Lab 3 

Sea Ice Dynamics 

Discovering Local Data 

Lab 4 

Land Ice Thermodynamics 

A Bird's Eye View: Exploring your Region 

Lab 5 

Glacier Dynamics 

It's all Connected: Global Circulation 

Lab 6 

Climate History & the Cryosphere 

Air, Water, Land & Life: A Global Perspective 

Lab 7 

Future of the Cryosphere 

A Year in the Life of the Earth System 


could be made to the EarthLabs curriculum materi¬ 
als. This helped to ensure that materials incorpo¬ 
rated the research evidence collected about student 
learning. 

(6) Iteration. Opportunities for revision and re-testing 
of materials were incorporated during ongoing 
studies to increase the robustness of efficacy testing 
of curriculum materials and student learning. 

In the described project, the EarthLabs research and the 
evaluation efforts were conducted by the same unit/team 
and were complementary/overlapping activities. However, 
often, research and evaluation are distinct efforts conducted 
by different parties. For instance, in an ongoing EarthLabs 
project, the research team is specifically focusing on the 
student learning outcomes and addresses research questions 
as a result of the designed curriculum, providing robust 
research evidence to modify generated curricula. A separate 
evaluation team aims to assess the program as a whole; 
specifically, they collect information about teacher profes¬ 
sional development activities and the curriculum develop¬ 
ment process, and they serve as unbiased third-party 
reviewers to the research design and findings. Evaluators 
in the ongoing project will utilize the summative student 
research results, in combination with the other project data, 
to evaluate the overall programmatic effort. In essence, 
research and evaluation efforts are uniquely tailored to the 
individual projects; however, in general, research efforts 
often address specific research questions, and evaluation 
efforts often address the project's programmatic goals— 
although a combination can exist, such as the case of the 
work presented here. 

Assessment 

Analysis of the efficacy of a complex curriculum requires 
multiple qualitative and quantitative assessments. To this 
end, we developed a process that incorporated multiple 
levels of data collected from a wide array of potential users, 
including teachers, high school students, and external users. 
This information was analyzed and then reported back to 


curriculum developers for consideration in future curriculum 
revisions and new development. In general, the use of two 
rubrics (which are further described below) for pre/post¬ 
assessments, thematic analysis of interview and open-ended 
responses, and the use of descriptive statistics and paramet¬ 
ric or nonparametric tests were employed during analysis of 
the results. Validity and reliability were established through: 
alignment of assessment questions with the curriculum 
goals, teacher, expert, and curriculum developer review of 
assessment instruments and subsequent modifications, and 
inter-rater reliability calculations on qualitative measures. 
Specifics about each of the instruments and methods 
employed with each student population, including the 
validity and reliability as appropriate, are described in further 
detail below. 

Student Assessments 

High school students completed assessment surveys for 
both the ESS and Cryosphere modules before and after 
classroom implementation (see supplemental materials). 
Pre/postassessments included four to six open-ended 
response questions that were aligned to curriculum learning 
goals and materials, six demographic items, and one self- 
confidence item for each question set. Responses were 
scored using a conceptual understanding rubric (score 0-5; 
Table II) for all questions on the ESS module, and a systems 
understanding rubric was used for all questions on the 
Cryosphere module and one question on the ESS module 
(Table III). For the conceptual understanding rubric (Table 
II), scores of 0 to 5 were made based on the number of facts, 
connections made, and misconceptions present in student 
responses. For the systems understanding rubric (Table III), 
student responses were parsed into actions and processes 
(usually represented by the use of a verb), inputs and outputs 
(usually represented by the use of a noun), and connections 
between them (usually represented by the use of a 
conjunction). Statistical analyses of pre/postcourse differ¬ 
ences were conducted using a paired Student's f-test when 
assumptions of the test were satisfied, including normality of 
the data and homogeneity of variance; otherwise, a 
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TABLE II: Explanation and scoring system for the conceptual 
understanding rubric. 1 


Level 0 

Simple restating of the question. 

Level 1 

Statement of a single correct fact. 

Level 2 

Statement of multiple correct facts. 

Level 3 

A. Statement of multiple correct facts, with a single 
connection between facts. OR B. Statement of 
multiple correct facts, with multiple connections 
between facts. Misconceptions equal to or dominate 
over scientific conceptions. 

Level 4 

Statement of multiple facts, with multiple 
connections between facts. Misconception(s) present, 
but scientific conceptions dominate. 

Level 5 

Statement of multiple facts, with multiple 
connections between facts. Misconceptions not 
present within a story that is cohesive; 
misconceptions about concepts outside of the core 
message may be present. 


1 Misconceptions may or may not be present in levels 0-3. 


Wilcoxon signed ranks test was used for nonparametric 
analysis. A nonparametric two-tailed Spearman's correlation 
analysis of student self-reported pre/postcourse confidence 
and overall scores was also completed. Inter-rater reliability 
was conducted with 10% of the student responses scored by 
two researchers, resulting in a minimum of 90% inter-rater 
agreement for both surveys. All disagreements were 
discussed and resolved, allowing a single researcher to 
continue coding all remaining responses. Experts in climate 
science also responded to each questionnaire, providing a 
basis for comparing student responses to those of expert 
scientists. 

Classroom observation offered information about stu¬ 
dent interaction and engagement with the EarthLabs 
materials. This method provides critical feedback about 
students' time-on-task and offers supporting evidence that 
their content knowledge was likely strengthened by their 
experience with the implemented curriculum materials. 
Classroom observers used a formal protocol (see supple¬ 
mental materials for instrument) to record student time-on- 
task while engaging with the EarthLabs curriculum and were 
trained to use the classroom observation tool to help ensure 
internal consistency (modified from Stallings, 1980). Ob¬ 
servers recorded what students were doing as they worked 
in groups (e.g., organizing, listening, and discussing, as well 
as time off-task) every 5 min for a 30 min period. Average 
student time-on-task in each category was then determined 
over the entire observation duration. The instrument was 
reviewed by a team of researchers and piloted by multiple 
observers to ensure a 90% or greater inter-rater reliability 
between observers. 

Teacher Feedback 

At the end of the classroom implementation year, 
teachers were also interviewed via telephone for the purpose 
of identifying opinions about what did and did not work, 
how students responded to the curriculum, and whether 
teachers believed the learning goals of the curriculum were 
met. Interviews were semistructured with predetermined 
questions, and probes were added dynamically in reaction to 
interviewee responses. Interviews lasted 30-45 min and 


TABLE III: Explanation and scoring protocol for the systems 
understanding rubric. 



Symbol 

Description 

Points 

Arrow 

Actions/Processes 
(Usually a verb) 

1 point 

Boxes 

Inputs/Outputs 
(Usually a noun) 

2 points 


Connections 

s 

Between 

Actions/Processes 

or 

Inputs/Outputs 
(Usually a conjunction) 

3 points 



were conducted in the presence of a moderator and a note- 
taker. Teacher responses, including direct quotes and 
analysis of themes, were used to inform the curriculum 
developers of teachers' perspectives of the EarthLabs 
classroom implementation. 

External User Assessments 

Eye-tracking data provide a window into the nature of 
student engagement with online materials. This is an 
important step in the materials development process, 
particularly for labs that contain significant interaction with 
online environments. As noted, the EarthLabs modules are 
designed for advanced high school students; eye-tracking 
data were collected from a population of entry-level college 
undergraduates. The eye movements of six to seven students 
were tracked and recorded for each lab within the Cryosphere 
and ESS modules. Participants were told to complete the 
module as though it were a class assignment and were given 
as much time as they needed to complete the task. After 
completing the experiment, participants were asked to 
discuss their experience with the EarthLabs site and 
comment on what they found most/least useful. Both eye¬ 
tracking data and participant comments were used to make 
recommendations for lab revisions. 

Research Context and Participants 

Human subject research approval was obtained by the 
appropriate institutional review boards, and participant 
consent was obtained for this research project. Information 
about each study population (teachers, students, external 
users, and experts) is included in Table IV. 

Students 

In total, 205 ninth- to twelfth-grade students partici¬ 
pated in the EarthLabs classroom implementation (Table IV). 
Six of the students were enrolled in a chemistry course, 12 
were in an environmental science course, 24 were in an 
astronomy course, 91 were in an Earth and space science 
course, and 72 were in either a regular environmental 
science course or an advanced placement (AP) environmen¬ 
tal science course. The majority of these students (n = 163) 
completed the ESS module, and the remaining students (n = 
42) completed the Cryosphere module. Most of the students 
were from the same school district in central Texas, although 
some were from other locations throughout the state. 
Student paired responses to questionnaires were analyzed, 
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TABLE IV: Demographics of participant populations. 


Participant Type 

Number 

Gender (Male/Female) 

Ethnicity 1 

Teacher 

7 

1/6 

C: 6; H: 1 

Student: 9th-12th grade 

205 

92/111 

C: 87; AA: 27; H: 78; A: 11; NA: 1; O: 1 

External users— Cryosphere 

26 

12/14 

C: 20; AA: 2; A: 4 

External users— ESS 

23 

6/17 2 

C: 18; AA: 3; A: 3; NA: 1 

Experts 

3 

2/1 

C: 3 


*C = Caucasian; AA = African-American; H = Hispanic; A = Asian; NA = Native American; O = other. 
2 One male is coded as AA, A, and C. 


with missing pre- or postcourse responses excluded from the 
data set. 

Teachers 

Seven high school educators, who taught the students 
described in the previous section, in Texas, implemented the 
EarthLabs curriculum in a chemistry, environmental science, 
astronomy, Earth and space science, or AP environmental 
science public high school classroom (Table IV). Seven 
classrooms were observed, with a total of 205 students 
completing pre- and post-tests from these classrooms. 

External Users 

In all, 49 entry-level undergraduates at a large Midwest¬ 
ern institution engaged in 1 h instructional sessions, where 
26 completed one of the labs in the EarthLabs Cryosphere 
module, and 23 completed either Lab 5 or Lab 6 in EarthLabs 
ESS module (Table IV). These students were 18-20 y old and 
were enrolled in an entry-level college geography/geosci¬ 
ence course or had taken no Earth-related science course 
beyond high school. 

Experts 

Three experts that held a doctoral degree in a geoscience 
field and were actively conducting relevant geoscience 
research were recruited to complete assessments in order 
to have a comparison for a likely maximum score we could 
expect from our instruments. The experts were recruited 
from a large geoscience department at a southern U.S. 
university and a national U.S. research laboratory (Table IV). 

RESULTS 

Student Results 

Classroom Observations 

Results from local classroom observations indicate that 
students were engaged with the EarthLabs materials more 
than 98% of the time. The majority of student time was 
spent working on the online curriculum (61%), engaging in 
hands-on activities (18%), discussing the materials (17%), or 
organizing materials (16%). We take these on-task results as 
strong indication that changes in students' conceptual and 
systems understanding and confidence levels are likely a 
result of the classroom instruction. 

Conceptual Understanding 

Students completing the Cryosphere module (;n = 42) 
exhibited significant increases in general awareness of the 
existence and underlying concept of the cryosphere when 
asked to "define and describe the cryosphere." Pre/postcourse 


conceptual understanding scores indicate significant improve¬ 
ments. Lor example, one student progressed from, "I don't 
know, sorry," to, "The sphere with ice, snow, sleet, etc." after 
instruction. Overall, 42% of students responded, "I don't 
know," prior to instruction, whereas only 18% responded, "I 
don't know" postinstruction. Additionally, over 57% of 
student postinstruction responses included the word "ice," 
whereas only 33% of pre-instruction responses included the 
word "ice." However, postinstruction responses indicate that 
numerous misconceptions persisted despite instruction, with 
some students indicating that the cryosphere is: "The ice-shelf 
biome" and "Where vegetation occurs." The first student does 
in fact understand that the cryosphere has something to do 
with ice on Earth; however, they relate it to a "biome" and 
exclude land ice or snow, showing that there may be confusion 
about what a biome is versus an Earth system. The second 
student seems to think that the cryosphere is a place on Earth 
that defines where vegetation occurs, illustrating a lack of 
understanding of what comprises Earth's cryosphere. Cumu¬ 
lative scores on the assessment as a whole also showed overall 
pre/postinstruction gains in classrooms that implemented the 
entire ESS and Cryosphere modules (pre-instruction M = 4.17, 
postinstruction M = 6.18, p < 0.05; Fig. 1). 

For the ESS module, results of the conceptual under¬ 
standing rubric show that students improved on all items 
from pretest to post-test (Fig. 2). The highest gains occurred 
on question one (Fig. 2), with mean pre-instruction scores 
less than 0.10 and postinstruction scores greater than 0.70. 
Analyses of example responses to specific items illustrate 
how the conceptual knowledge rubric was used to show 
specific patterns in the data. Students were asked to 
"describe and define Earth system science in your own 
words." Analysis of the responses using the conceptual 
knowledge rubric (Table II) indicates statistically significant 
differences (p < 0.05) between pre- and postinstruction 
scores (pre-instruction M = 0.067, postinstruction M = 
0.75). For example, one pre-instruction student response, 
"the research of the way that the Earth operates... this show 
us how the Earth is changing as a whole," is less complete 
and nuanced than the same student's postinstruction 
response, "The work of geography such as biosphere, 
pedosphere, atmosphere, and hydrosphere used to find 
information and solve facts for future use and help us find 
out more about the way our planet works." Students were 
also asked to, "Please look at the two images provided [one 
of global precipitation in August and a second of global 
precipitation in February] and explain what makes the two 
images similar/different" (see supplemental materials). 
Analysis of pre/postinstruction conceptual understanding 
scores for this question indicate statistically significant (p < 
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Earth Systems Cryopshere Crysosphere Crysosphere 
Module Module (All) Module (Some) Module (Few) 


■ Pre-instruction 
Post-instruction 


FIGURE 1: Average pre/postinstruction conceptual understanding scores for the Earth Systems Science (ESS) module 
and average pre/postinstruction systems understanding scores for the Cryosphere module. The ESS module is 
aggregated for all classrooms. Each classroom is shown for the Cryosphere module to show effect of different time 
exposures on learning outcomes. All = the classroom that implemented all labs. Some = the classroom that 
implemented some of the labs. Few = the classroom that implemented only a few or limited labs. 


0.05) improvements, with postinstruction scores (M = 1.09) 
almost double the pre-instruction scores (M = 0.65). 
Another example shows how a student progressed from 
explaining, "It rains more in February" (pre-instruction), to, 
"It is very hot over the equator so water from the oceans 
evaporates up to form clouds and then is carried over the 
continents by wind and then rain over the continents." This 
second explanation is much more nuanced and comprehen¬ 
sive. Postinstruction, approximately 35% of the students 
identified seasons as a cause for change in precipitation, 13% 
deciphered differences between the Northern and Southern 
Hemispheres, 33% identified the equator in the provided 
map, 17% linked color intensities in the map to precipitation 
amounts, and 2% provided other scientific reasons for 
differences between the provided precipitation images. 


Systems Understanding 

For the Cryosphere module, results show that students 
engaged in all of the module's labs had significant (p < 
0.05) pre/postinstruction systems understanding learning 
gains on individual assessment items after implementation 
(pre-instruction M = 2.53; postinstruction M = 3.57). Three 
of the four pre/postinstruction assessment questions 
showed improvements for students that participated in all 
aspects of the Cryosphere module (Fig. 3). Examples of a 
student and an expert response are illustrated in Figure 4 to 
show how the systems rubric was applied in the Cryosphere 
module. For this example, the question asked students to 
"explain why the ice extent in each of the images are 
different" upon viewing a satellite image of sea-ice extent 
over a 270 d period (see supplemental materials). Analysis 



FIGURE 2: Student average pre/postinstruction conceptual understanding performance on individual items on the 
Earth Systems Science module assessment. 
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FIGURE 3: Student mean pre/postinstruction systems understanding performance for individual items on the 
Cryosphere module assessment for each implementing classroom: (a) the classroom that implemented all labs, (b) the 
classroom that implemented some labs, and (c) the classroom that implement few labs. 
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The images below show ice extent ice (in white) in the Northern Hemisphere over a 270 day period. 
Notice that the approximate location of the Unites States (USA) is labeled in each of the figures. 
Answer the following questions to the best of your ability. 



^y 1 Day 90 Day 180 Day 270 

Explain in your own words why the ice extent in each of these images is different. 


Student response: "The hemisphere is warming, causing ice to melt." 



Expert response: "The images show the sea ice extent through the seasonal cycle. In the winter the 
sea ice extent would be the greatest. This is because there is no or little solar radiation over the region 
at that time of the year, causing air temperatures to be very low, resulting in a growth and expansion 
of sea ice at the surface of the ocean... The image for day 180 shows the sea ice extent reduced from 
the image for day 90 so that image shows the sea ice extent in the spring." 


Solar Radiation 


drops/ is low 


Air Temperatures i 


drop 


Sea Ice 


grows 


Sunlight 


is present 


AirTemperaturesi 


rise 


Ice Temperatures' 


reach melting 


Ice 


melts 


36 


FIGURE 4: An example open-ended Cryosphere pre/postinstruction assessment question and student and expert 
responses as scored by the systems rubric in response to, "Explain why the ice extent in each of the images is 
different." 
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FIGURE 5: Pre/postinstruction cumulative student scores for all questions on the Cryosphere module using the 
systems rubric. 


of the distribution of systems understanding scores for all 
questions indicates that cumulative scores were below 25 
points in pretests but reached as high as 30 on post-tests 
(Fig. 5). Over approximately half of the student respondents 
scored higher than 20 points in the postassessment. Pre- 
and postinstruction systems thinking scores differed 
significantly on a Wilcoxon signed ranks test (p = 0.08, Z 
= —1.74). Interestingly, students who completed only a 
portion of the labs exhibited no statistically significant 
improvements postimplementation (p > 0.05), whereas 
those engaged in all seven labs of the modules and those 
who spent nine in-class hours over the course of 4 weeks 
on these materials showed statistically significant improve¬ 
ments postimplementation (p < 0.05). 

On the ESS module assessment, increased systems 
understanding is illustrated by student responses to "draw 
and label arrows to represent ALL of the important processes 
that move or change energy, water, or chemicals in this 
region" (Fig. 6). Analysis indicates that students showed 
significant increases in the number of arrows drawn (pre < 3, 
post > 5). Furthermore, paired Student's t-test analysis 
indicates statistically significant differences (p < 0.05) on 
pre/postinstruction mean scores using the systems rubric 
(Table III; pre-instruction M = 8.72, postinstruction M = 
12.81). When expert geoscientists (n = 3) were asked to 
complete the same task, the average number of arrows drawn 
was 10, and an average score of 34 was achieved (Fig. 6c). 

Confidence Scores 

Finally, all students (n = 205) were asked to reflect on 
their confidence with regard to understanding the material, 
using a Likert scale (1-4) to rate their ability to answer 
questions about the EarthLabs content. Self-assessed pre-/ 
postinstruction student confidence on the ESS module 
increased from an average of 1.20 to 1.94. Similar results 


were observed for the Cryosphere module, where students 
either maintained or increased their confidence level from 
1.94 to 2.90. A Student's f-test showed pre- and post- 
instruction confidence score increases were statistically 
significant for both modules ( Cryosphere p = 0.000, ESS p 
= 0.004). A Spearman's correlation analysis showed that 
student confidence scores and student performance were 
significantly (p < 0.05) positively correlated (corr. coeff. = 
0.655). 

Teacher Postimplementation Interviews 

Seven teachers were interviewed by phone postimple¬ 
mentation about their perceptions of student learning. 
Teachers were asked whether they believed their students 
better understood specific objectives (e.g., physical change 
over multiple time scales, dynamic interactions among 
Earth's systems, relevance to students' own lives) after the 
implementation of the EarthLabs modules. 

With relation to time scales of cryospheric change, one 
teacher responded, 

"Interestingly, enough kids keep bringing in the Deadliest 
Catch [a reality TV show on commercial fishing] because 
they are always showing ice over the Bering Sea and ice 
coming down depending on time of year. So kids caught onto 
that idea 'over time,' because they could relate it to the 
fishermen and how the sea ice moves and changes and how 
quickly it can move and change." 

Another teacher indicated that their students gained a 
partial understanding: 

".. .they got part of change and fluctuations. Larger time- 
scales. . .understood more of seasonal aspects, not sure how 
many got change over longer period of time ." 
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FIGURE 6: Pre- and postinstruction EcirthLabs ESS drawing illustrating systems understanding (a) before and (b) after 
implementation, and (c) an expert geoscientist drawing in response to, "The following diagram represents a region in 
the continental United States. Draw in and label arrows to represent ALL of the important processes that move or 
change energy, water, or chemicals in this region." 


When teachers were asked about whether their students 
better understood dynamic interactions among Earth's 
systems, one teacher responded, 

"I think they got a better idea of it, of how things are 
interrelated, how a change in one place can end up being a 
change in another.. .They were starting to see that." 

A second teacher provided a more nuanced view, stating 
that their students did have 

"Some [improvements] in terms of the Earth's system 
[components], but maybe not the bigger [Earth] system..." 

Yet another teacher replied, 

"In AP environmental course we try to look at Earth as 
system.. .the EarthLabs tied in the big picture much better 
fhan I have been able to do in the past." 


When teachers were asked whether students have a 
concept of the relevance of complex Earth systems and the 
importance of the cryosphere to their own lives, one teacher 
responded, 

"I believe so.. .all mentioned to me they actually enjoyed 
doing it because they did learn something new and it was 
interesting for them because it was so new to them. Kind 
of new for all of us. One of those areas that has been 
neglected in Earth Science—does play such an important 
role.. .global warming issue has brought it to the 
forefront." 

At the same time, some teachers did note that 
students had some troubles relating the cryosphere to 
their own lives in Texas; many students did not have 
firsthand experience with frozen precipitation. These 
teachers felt students were "too far removed" and unable 
to see their personal "connection" to snow and ice. 
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Overall, teachers commented that the strengths of the 
materials included the use of many visuals, movies, and 
graphs that engaged the students and forced them to think 
about the problem. One teacher commented that he/she 
"would enjoy having a lot more resources like [Earth- 
Labs]." Teachers largely felt that the materials supported 
students' ability to learn about the cryosphere and Earth 
systems science. 

External Users 

Eye-tracking data provide information about the areas of 
the online curriculum that attract novice viewer gaze, and 
hence with which viewers are likely engaged, and these data 
can be coupled with interviews. Recall that all data discussed 
here were collected from non-science-major college stu¬ 
dents (n = 49) who completed one lab only. Although too 
rich to discuss in their entirety here, we provide examples 
from the Cn/osphere module's "Sea Ice Dynamics" lab to 
illustrate the value of this research technique for informing 
curriculum design. 

A heat map aggregating the gazes of multiple viewers 
(n = 6; Fig. 7) indicates that viewers are both engaged by 
the core text in the "Sea Ice Dynamics" lab and are not 
distracted by ancillary objects (e.g., heading, table of 
contents). However, students were mainly engaged with 
the text and not the images. Alternatively, a comparison of 
gaze paths for two individuals, showing the path students' 
eyes took across a page (Fig. 8), indicates that the amount 
of attention students pay to external Web links varies 
significantly. Figure 8 illustrates the range of engagement 
observed via eye tracking; as noted, most students 
exhibited a high level of engagement (as in Fig. 8b) at the 
start of the lab with decreasing engagement over time (as in 
Fig. 8a). 

Overall, data indicate that students spent between 23 
and 32 min completing the lab, engaged less with images 
than with text, and had difficulty engaging with some 
materials posted on external sites, such as graphs depicting 
change in climate indices over time. Eye tracking of student 
interactions with other labs yielded similar results, sug¬ 
gesting that: (1) labs are sufficiently engaging for students; 
and (2) materials hosted by external sites (e.g., National 
Aeronautics and Space Administration [NASA], National 
Oceanic and Atmospheric Administration [NOAA]) may 
need to be modified by the currculum team before students 
can engage effectively with them. 

The nature of participant interactions with pages and 
elements within pages varied greatly across individuals and 
over time. We note that the first page of each lab attracts 
significant attention to all elements within the page. 
Attention to text elements in subsequent pages was 
significantly reduced, suggesting increased fatigue or 
waning interest as participants worked through the labs. 
Participants also did not always utilize external sites linked 
from the EarthLabs modules, and viewing of external sites 
was much more cursory than engagement with the Earth¬ 
Labs pages themselves. Finally, participants were asked to 
discuss their experience with the EarthLabs modules and 
comment on what they found most and least useful. In 
general, participants found charts, graphs, and questions 
embedded in text most useful (64%), followed by videos 
and external Web sites (32%). Only one student indicated 
that the text components of the lab were helpful for 


learning about the climate system. This low level of 
valuation coupled with waning attention to text over time 
suggest that reduction of text might be helpful for engaging 
students over the amount of time needed to complete the 
labs. 


DISCUSSION 

The collaboration of curriculum developers, education 
researchers, external evaluators, professional development 
specialists, teachers, and their students provided an 
exciting venue for developing and assessing the EarthLabs 
climate change modules as an exemplar of evidence-based 
online curriculum development. This work shows the 
results of student learning during classroom implementa¬ 
tion of the EarthLabs online modules. It also provides 
feedback and research-quality results to the curriculum 
developers about how further EarthLabs materials should 
be refined. 

How Does the Online EarthLabs Curriculum Assist 
Students in Developing Understanding of Temporal 
and Spatial Dynamics and System Interactions of 
Climate? 

As a result of the EarthLabs implementation, students' 
conceptual and systems understanding significantly im¬ 
proved, as evidenced by significant changes in pre/post¬ 
instruction assessment scores and systems understanding 
scores. The complex interplay of climate acting over 
multiple Earth systems requires an understanding of the 
temporal and spatial nature of climate system dynamics. 
The attention paid in EarthLabs to concepts of scale within 
specific climate phenomena, such as in the Cryosphere 
module, afforded students many opportunities to wrestle 
with changes that are observable over human life spans, as 
well as those that occur over periods of time, or over scales, 
that are outside human perception. According to the 
student learning goals (Table I) for these modules, both 
modules focus on change over time; however, the ESS 
module focuses more on connections between parts of a 
system. 

Despite marked improvement, students did not reach 
mastery levels on par with expert responses. This lack of 
mastery is expected for high school students in early stages 
of learning about systems. In addition, the cognitive 
hierarchical structure of systems thinking that includes 
"stages" or cognitive steps in development of the next 
higher-order thinking skills (Assaraf and Orion, 2005) 
predicts a disparity between experts and novices in 
grappling with the causal behaviors and functions of 
complex systems (Hmelo-Silver et al., 2007). We would 
not expect these students to reach the level of experts, of 
course, but would like to note that this improvement in 
systems thinking ability aligns with expectations of the 
systems crosscutting concept recently codified in the Next 
Generation Science Standards (Achieve, Inc., 2013). 
Despite a lack of mastery, it is important to note that 
the majority of students experienced significant changes in 
their understanding even after only one EarthLabs module 
exposure. Students also increased their content knowledge 
and confidence in engaging with complex climate concepts 
during the time frame of the implementation, where 
significant correlations between student performance and 
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FIGURE 7: Heat map of student (n — 6) attention to the "Sea Ice Dynamics" page of Lab 3 in the EarthLabs Cryosphere 
module. Gray clusters indicate areas of viewer attention. Darker areas represent more attention. 


confidence were measured. Teachers also reported that 
students achieved many of the EarthLabs learning goals. 
Furthermore, interviews and feedback from teachers 
suggest that more intentional teaching about temporal 
and spatial changes in the Earth system is needed in high 
school curricula, identifying that EarthLabs modules may 
help build capacity for environmental science courses to 
meet student learning needs. 


How Do Users Engage with and Navigate the Online 
EarthLabs Curriculum? 

Classroom observations combined with eye-tracking 
measures of external users indicate that the EarthLabs 
modules were engaging to users. However, some modifica¬ 
tions in page content could be made to increase the long¬ 
term engagement of users as they navigate through the 
modules. For example, these data indicate that students do 
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FIGURE 8: Two individual gaze paths for Blue Marble site (external link on EarthLabs ESS module). Path (a) is cursory 
and is typical of most participants, and path (b) is a participant with significant attention. 
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TABLE V: Synthesis of the design-based research theory, the EarthLabs research-curriculum design cycle, and example activities 
and outcomes. 


Design-Based Research Theory 

EarthLabs Research-Curriculum Design 
Cycle 

Example EarthLabs Curriculum and 
Research Activities and Outcomes 

It should design and test a 
significant intervention. 

Create. Curriculum developers designed 
explicit student learning goals and developed 
curriculum level assessments while online 
materials were generated. 

EarthLabs Cryosphere and ESS modules and 
learning goals were developed (see Table I). 

It should be grounded in theory to 
inform the research design. 

Test. Researchers developed appropriate 
research questions and coupled research 
quality assessments (based on existing 
literature) before and during the curriculum 
development process and worked with 
developers to ensure that assessment content 
aligns with EarthLabs curriculum goals and 
materials. 

EarthLabs research questions were developed: 

How does the online EarthLabs curriculum 
assist students in developing understanding of 
temporal and spatial dynamics and system 
interactions of climate? 

How do users engage with and navigate the 
online EarthLabs curriculum? 

It should be interactive and flexible, 
where designers are involved in the 
process and work together. 

Revise. Discussion between researchers and 
curriculum developers occurred continuously 
to generate feedback and continual revision 
of student learning goals, research questions, 
and develop assessments before curriculum 
implementation. 

Modifications to initially developed 
assessments were made through feedback 
from curriculum developers to ensure 
alignment to learning goals and "big ideas." 

It should be integrative, where 
mixed method approaches are used 
in the research. 

Implement. Appropriate qualitative and 
quantitative approaches were employed to 
address the research questions generated in 
order to triangulate data, enhance robustness 
of interpretations, and maximize data 
collection opportunities. Validity and 
reliability measures were implemented. 

EarthLabs final assessment instruments were 
developed (see supplemental materials). 

It should be practical, where the 
research refines both theory and 
practice. 

Revise. Collected data and results were 
shared with curriculum developers well 
before upcoming implementation stages so 
that adequate modifications could be made to 
the EarthLabs curriculum materials. 

Dynamic and embedded animations that assist 
students in visualizing flow of energy and 
matter in a variety of Earth systems were 
made. 

Figures and videos were directly embedded; 
text to describe images was added; direct 
references to images within the text body were 
added; and larger text blocks were shortened. 

It should be contextual, where the 
research results are connected with 
the design process and have a 
practical impact on instructional 
practice within specific settings. 

Emphasis on implementation of all labs with 
suggestions for how best to implement 
portions of the modules throughout the year 
to meet NGSS and state science standards. 

An iterative cycle of analysis, design, 
implementation, and redesign is 
used. 

Iteration. Opportunities for revision and re¬ 
testing of materials were incorporated during 
ongoing studies to increase the robustness of 
efficacy testing of curriculum materials. 

Findings from EarthLabs research have been 
incorporated into the Cryosphere and ESS 
modules as well as ongoing studies in the 
development of EarthLabs Climate modules. 


not always click on external links or watch videos as 
expected by curriculum developers. Additionally, the data 
indicated that students do not always spend a significant 
amount of time looking at images and attention may wane 
in longer text segments. Eye-tracking studies of these new 
design elements are ongoing. 

EarthLabs Curriculum Changes as a Result of the 
Research Findings 

The research outcomes from the Climate and the Cryo¬ 
sphere and the Earth System Science modules were reported 
back to the EarthLabs curriculum developers and have been 
incorporated into the Climate and the Cryosphere, Climate and 
Biomes, and Climate and the Carbon Cycle modules, which are 
currently undergoing revisions, and another cycle of revise 
and test is under way. Table V illustrates how the design- 


based research theory from the literature was incorporated 
into the EarthLabs research-curriculum design cycle, where 
example outcomes and research-informed curricula modifi¬ 
cations are shown. 

Research showed that scaffolding was needed to help 
students to view images by incorporating more references to 
the images in the text and embedding images in text where 
possible. In response, curriculum developers reworked text 
pages to embed figures and videos, including text directly 
into the EarthLabs pages instead of as an external link, 
assisting viewers to better utilize these features of the 
curriculum. Developers have integrated additional text that 
describes images; incorporated more direct references to 
images within the text body; and separated and shortened 
larger text blocks. 
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Additional findings suggest that the EarthLabs curricu¬ 
lum may need to provide increased scaffolding to students in 
regard to developing higher-order thinking skills, specifically 
in regard to making connections between systems, in order 
to assist students to reach a mastery level of understanding 
and to continue to build their confidence. In response, 
curriculum developers have built their own dynamic 
animations that are embedded into the EarthLabs modules 
that assist students in visualizing flow of energy and matter 
in a variety of Earth systems. 

Our findings also suggest that those students exposed to 
all labs of the Earthlabs Cryosphere module had greater 
learning gains than those that only completed a portion of 
the labs, while multiple exposures to a series of EarthLabs 
modules in an entire semester-long course will likely result 
in the strongest conceptual changes. In response, the 
EarthLabs professional development efforts have emphasized 
that teachers implement all labs in a given module in order 
to achieve the intended learning outcomes. In school 
settings, it can be difficult to spend 2 to 3 weeks on each 
module, so suggestions have been made to teachers by staff 
and teacher leaders in how best to implement portions of the 
modules throughout the year to meet NGSS and state 
science standards over a continual basis in order to cover all 
labs in an EarthLabs module and maximize potential for 
students to reach the intended learning outcomes. 

CONCLUSION 

Careful development of curriculum materials is impor¬ 
tant for effective instruction, and particularly for those 
domains that will influence the way in which society grows 
and evolves in response to social, economic, and environ¬ 
mental needs. This study illustrates how research-driven 
online curriculum that allows learners to navigate near real¬ 
time data representing spatial and temporal changes and 
complex Earth and climate interactions can be conducted. 
We note that it can be difficult to engage in a full cycle of 
create-test-revise-implement-revise for any one curriculum 
due to limitations in personnel, time, and funding. In the 
EarthLabs case, the collection of curricular efficacy data was 
used to revise existing materials and inform the development 
of new materials in ongoing, follow-on projects. In an age 
when materials are widely disseminated via the internet, we 
as scientists and educators must be diligent in ensuring that 
we are creating and sharing materials proven to be effective. 
As such, implementations of EarthLabs in classrooms by 
teachers and their students have been tracked during the 
research activities highlighted in this paper. Results have 
indicated that the online EarthLabs modules improve student 
understanding of temporal and spatial dynamics, and Earth 
system complexity, and that the learners are engaged with 
the curriculum as evidenced by classroom observations and 
eye-tracking experiments. The quantitative and qualitative 
findings from the research conducted in this study were 
employed to make recommendations for curricular improve¬ 
ments for the next round of classroom implementations. The 
EarthLabs curriculum-research model is an example of how 
research can be used to inform curriculum development, and 
it serves as a model for best-practice in online curriculum 
design and research. 
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